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METHOD AND APPARATUS FOR DE-INTERLACING VIDEO IMAGES 

Related Applications 

This application claims the priority of the corresponding provisional application, 
Serial No. 60/192,294, filed March 27, 2000. United States Patent application Serial No. 
(H. Jiang Case 1 1) was filed concurrently herewith. 
Technical Field 

This invention relates to video images and, more particularly, to the conversion of 
an interlaced field to a progressive frame. 
Background of the Invention 

Arrangements are known for converting interlaced video fields to progressive 
video frames through interpolation of so-called missing lines. One known arrangement 
of particular interest is disclosed in U. S. Patent 4,989,090 issued to J.J. Campbell et al. 
on January 29, 1991. This arrangement includes a video pixel interpolator that generates 
so-called interpolation pixels from incoming image pixels for use in a television image 
scan line doubler. The interpolator includes a temporal median filter that generates an 
interpolation pixel by selecting the median one of a plurality of temporal pixel samples. 
The reason for using the temporal median filter is so that a switch over from frame 
interpolation to field interpolation can take place at a higher motion threshold for the 
pixel. The switch over at a higher motion threshold is necessary in the Campbell et al. 
apparatus because of a high noise level there are no gaps in the motion values between 
moving and still pixels. Consequently, it would be difficult to determine whether or not 
the image at the pixel depicts motion, but for the use of the temporal filter. 
Unfortunately, the use of the temporal median filter in the Campbell et al. apparatus has 
only minor affects in the result. The purpose of using the temporal median filter is to 
allow the use of field interpolation even during higher motion values so that no 
objectionable aliases will be caused in the image by frame interpolation. However, at 
motion values when objectionable aliases would occur, the use of the temporal filter in 
the Campbell et al. apparatus still yields frame interpolation and, therefore, it does not 
remove the objectionable aliases. 
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Summary of the Invention 

These and other problems and limitations of prior de-interlacing arrangements are 
overcome by determining the motion at each missing pixel and, then, interpolating the 
missing lines to convert an interlaced field to a progressive frame. The interpolation 
5 employed for luminance is determined through motion detection. If motion is detected in 
the image, field based interpolation is used and if no motion of the image is detected, 
frame interpolation is used. 

Specifically, the interpolation is determined by employing a motion metric. The 
motion metric at a missing pixel is defined by using a prescribed combination of pixel 
10 luminance value differences. A spatial median filter is then used to remove objectionable 
noise from the pixel luminance value differences and to fill in so-called "holes" in the 
image. Indeed, the spatial median filter can be considered as providing a measure of the 
overall effect of all pixels that make up the object of the image. 

In a specific embodiment of the invention, a nine point spatial median filter is 
15 used to filter the noise from the pixel luminance value differences while continuing to 
preserve the motion or the stillness of the image. 

In still another embodiment of the invention a look-up table is used to determine a 
"weight" parameter, i.e., blending factor, for frame based or field based interpolations. 

A technical advantage of the invention is that it makes a correct decision 
20 regarding the motion state of the image rather than merely providing a so-called "fix" for 
erroneous decisions. 
Brief Description of the Drawing 

FIG. 1 shows, in simplified block diagram form, details of a de-interlacer in 
accordance with the invention; 

25 FIG. 2 graphically illustrates missing lines in interlaced fields useful in describing 

the invention; 

FIG. 3 is a graphical representation of a number of fields useful in describing 
taking the luminance differences of pixels; 

FIG. 4 shows, in simplified form, a nine-point spatial median filter that may be 
30 employed in practicing the invention; and 
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FIG. 5 is a graphical representation of a look up table including weights, i.e., 
blending factors, that may be used in the interpolation employed in the invention. 
Detailed Description 

FIG. 1 shows, in simplified block diagram form, details of a de-interlacer in 
5 accordance with the invention. The process of de-interlacing is to interpolate missing 
lines in an interlaced image field. 

Specifically, an image to be de-interlaced is supplied to input 101 and, then, to 
smoothing filter 102, via bypass 103 to a terminal of controllable switch 104, field 
interpolation unit 105 and frame interpolation unit 106. Smoothing filter 102 is 
10 employed to remove or reduce the noise level of the incoming image to remove its 
adverse effects on a motion metric to be generated and may not be required in all 
applications of the invention. In this example, a simple 1-2-1 horizontal filter may be 
used for this purpose. It should be noted that the smoothing filter 102 is employed only 
to compute the motion metric. After the weights a are computed, as described below, 
15 smoothing filter 102 is by-passed via bypass 103 and controllable switch 104, and the 
subsequent interpolation is done on the original images. 

Briefly, FIG. 2 shows two interlaced fields where "X" indicates existing lines and 
"O" indicates missing lines useful in describing interpolation. 

Broadly, interpolation for luminance is effected by using motion detection. If an 
20 image is found to be still, frame based interpolation is used. That is, the luminance value 
of the missing pixel "C 0 " is taken to be the value at the missing pixel in the early field, 
namely, 

C 0 = C_, . This is realized in frame interpolation unit 106. 

If the image is moving, i.e., has motion, then field-based interpolation is used. 
25 That is, the luminance value of the missing pixel "C 0 " is taken to be the average of the 
luminance values of pixels in the same field above and below the missing pixel, 

namely, C 0 = ( N °* s °^ . This is realized in field interpolation unit 1 05. 

In general, the motion of an image is characterized by a quantity, i.e., weight or 
blending factor, a, where 0<a<l, and the interpolation is given 
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by,C 0 =a^y^ + (l-a)C_ 1 . This is realized in alpha blender 112 in conjunction 

with a blending factor a from look up table 1 1 1 and the above-noted expressions from 
field interpolation unit 105 and frame interpolation unit 106, 

The interpolation of chrominance is always field based. 
5 Motion detection is accomplished by taking the luminance value differences of 

pixels of prescribed fields via pixel difference unit 107, as shown in FIG. 3. In this 
example, to determine the motion for a missing pixel, five pixel luminance value 
differences are obtained by pixel difference unit 107 in accordance with prescribed 
criteria as follows: 

10 A C =\Q-C_ X \; 
*.=\N 0 -N_ 2 \, 

A,=|S 0 -S_ 2 |; 



A = 



N o +S 0 N_ 2 +S_ 2 



and 



2 

A b =|C_, -C_ 3 | 

15 In the above expressions, C, represents the luminance value of the corresponding pixel in 
field f {9 C 0 , N 0 and S 0 are in field f Q9 C.j is in field f_ {9 N_ 2 and S_ 2 are in field /_ 2 
and C_ 3 is in field /_ 3 . It should be noted that only four image fields are used in 

determining the pixel luminance value differences and, hence, the motion metric A . 

The desired pixel luminance value differences are low pass filtered via low pass 
20 filter 108 to smooth them and the filtered versions are supplied to motion detector 109. 

Motion detector 109 actually filters the pixel luminance value differences from 

pixel difference unit 107 to remove aliases occurring under motion conditions. 

Moreover, it should be noted that all the pixel luminance value differences noted above 

might not be used in determining the motion of the missing pixel. The motion metric A 
25 at a missing pixel may be defined by employing some combination of the obtained pixel 

luminance value differences, for example, by 

A = max(A c ,AJ. 
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Other combinations of the pixel luminance value differences may also be used to obtain 
the motion metric at the missing pixel, for example, 
A = max(A, ? min(A tt5 A i )) ) 

is employed in motion detector 109 in this implementation. Note that the use of 
5 min(A„, AJ reduces the spreading of spurious motion in a vertical direction of the image. 

It is also important to note that our implementation is significantly simplified because the 
motion values are computed directly from the pixel luminance value differences 
employing the minimum and maximum value choices. 

The effects of using other examples of combinations of pixel luminance value 
10 differences on the quality of images are now briefly discussed. To this end, motion 
metric A = max(A c ,AJ is considered the reference. All the following motion metrics 

will be compared with it. Indeed, this reference motion metric expression produces 
satisfactory results for most situations. 

Consider motion metric A = max(A c? A w , A s ) , This motion metric varies slightly 
15 from the reference and produces similar quality images. 

Consider motion metric A = max(A c ,min(A„, A J). This motion metric has the 
advantage of preserving very well the edge of a still region in an image. However it 
produces slightly more aliasing than the reference motion metric. 

Consider motion metric A = max(A c ,A B ,A A ,A 6 ) . This motion metric has the 

20 advantage of removing more aliasing. However, disadvantages are that it causes a 
delayed motion and requires more memory. 

Consider motion metric A = max(A nJ A 4> AJ. In motion metric 
A = max(A c ,A /J ,A i ), the computation of A c requires a delay of one field. This delay 
may cause the images to be out of synchronization with associated audio. Exclusion of 
25 A c avoids this problem. However, disadvantages are also that it causes a delayed motion 
and requires more memory. 

It should be noted that the order of spatial medium filter 110 and look-up table 
1 1 1 could be exchanged. 
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In this example, the motion metrics A are computed by motion detector 109, 
filtered via spatial median filter 1 10 and, then, a look up table 1 1 1 is employed to obtain 
the weight, i.e., blending factor, a for the frame-based interpolation in frame 
interpolation unit 106 or field-based interpolation in field interpolation unit 105. 
5 FIG. 4 shows, in simplified form, details of a so-called 9-point spatial median 

filter 110 that is advantageously used in practicing the invention. It is noted that the 
pixel luminance value difference is only a measure of the change in a single pixel. 
However, when considering whether an object in the image is moving or not all pixels of 
the object should be considered. The spatial median filter 110 can be thought of as 

10 measuring the overall effect of all pixels that make up the object. Additionally, since 
each individual pixel luminance value difference may be prone to random noise, use of 
spatial median filter 110 can also reduce the effects of the noise. 

Referring to FIG. 4, it is seen that the 9-points (i.e., motion metrics A) are 
arranged into three groups of three points each, namely, a first group including motion 

15 metrics a, b and c, a second group including motion metrics d, e and f, and a third group 
including motion metrics g, h and j. The first group is supplied to sorter 401, the second 
group to sorter 402 and the third group to sorter 403. The motion metric A values are 
supplied from motion detector 109. Sorters 401, 402 and 403 each perform a complete 
sort of their respective supplied groups, i.e., arrange the supplied motion metric values in 

20 either ascending or descending order. In the spatial median filter shown in FIG. 4 it is 
. assumed that the motion metric values are arranged in ascending order. That is, 
a 3 >a 2 > a x and so on for the other values. Note that a sorter of three values requires 

three comparisons. Thus, the three sorters 401, 402 and 403 perform nine comparisons. 
The median of each group is determined to be the middle value motion metric in the 

25 sorted group. The three medians from sorters 401, 402 and 403 , in this example, are a 2 , 
b 2 and c 2 , respectively, and are supplied to sorter 404. In turn, sorter 404 sorts the three 
medians a 2 , b 2 and c 2 . This requires another three comparisons. After sorting, the three 
medians a 29 b 2 and c 2 , are assumed to be arranged in ascending order and are designated 
X , /? and y , respectively, where X<f)<y. Now the nine points of median filter 1 10 

30 are reduced to five points by removing four points. The remaining five points include the 



6 



H. Jiang 10-20-9 

median of the nine points. This reduction is realized by first identifying the group of three 
values whose median is X . These values are labeled in ascending order as d ] <d 2 <d i . 
It is noted that these three values had been sorted in the prior sorting operations. 
Additionally, since d 2 is the median of the group, it has the same value as X . It can be 

5 shown that both d x and d 2 can be removed from the nine points. Now label the three 
values having y as its median in ascending order as f x <f 2 <f y . Again, it is noted that 
f 2 has the same value as y . It can be shown that the values f 2 and / 3 can be removed 
from the nine points. Thus, leaving five points including d 3 , f x and a group of three 
values having as its median that is labeled in ascending order as e x < e 2 < e 3 . These 

10 remaining five values are divided into two groups and further sorted. One group includes 
d z and e x that after sorting via sorter 405 are labeled in ascending order as g x < g 2 . This 

sorting requires only one comparison. The second group includes e 2 , e 3 and f x that after 

sorting via sorter 406 are labeled in ascending order as \ < \ < . This sorting only 

requires two comparisons because e 2 and e 3 have already been sorted. Of the remaining 

15 five values g x , g 2 , \ , and h$ , it can be shown that values g x and h$ can be removed, 
leaving values g 2 , \ and . These remaining three values are sorted via sorter 407 and 
labeled in ascending order as j\ < j 2 < j 3 . This sorting takes only two comparisons 
because values \ and have already been sorted. The median value of the group from 
sorter 407 is the median of the nine points and is value j 2 . 

20 It should be noted that if so-called pipelining is used in the median filter 110, only 

one three point sorter is required for sorters 401, 402, 403 and 404 because the prior 
sorted results are stored for use in the subsequent sortings. 

Moreover, the use of this unique spatial median filter 110 removes or reduces the 
effect of noise on the motion metric values without generating spurious "stillness" or 

25 motion. Furthermore, use of the spatial median filter in the invention enables the correct 
decision to be made regarding the motion state of an image rather than just providing a 
"fix" for erroneous decisions made in prior de-interlacing arrangements. 
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For further details of spatial median filter 1 10 see United States patent application 
Serial No. ( Hong Jiang Case 11) filed concurrently herewith and assigned to the assignee 
of this patent application. 

FIG. 5 is a graphical representation of a look up table including weights, i.e., 
5 blending factors, that may be used in the interpolation employed in the invention. In this 
example, the look up table is represented as a stretched sinusoidal curve, where a has 8- 
bit values. In certain applications, a may use fewer bits. It is noted that the curve shown 
in FIG. 5 has significant effects on the quality of the de-interlaced images. Shifting the 
curve to the left causes more pixels to be interpolated based on field, and therefore 
10 reducing aliasing. On the other hand, shifting the curve to the right may increase 
aliasing. 

Thus, the look up table of FIG. 5 yields the weight, i.e., blending factor, a based 
on the supplied median motion metric A output from spatial median filter 1 10, namely, 
median value j 2 , Then, the weights, i.e., blending factors, a are supplied to alpha (a) 
15 blender 112. It should be noted that theoretically either the spatial median filter 1 10 or 
the look up table 1 1 1 could be applied first to the motion metric A . 

In one example the blending factors for given motion metrics are as follows: 



Motion Metric Value Blending Factor 

20 0 0 

1 0 

2 0 

3 0 

4 23/255 
25 5 93/255 

6 170/255 

7 240/255 

8 1 (255/255) 

In this example, any motion metric value of less than 4 yields a blending factor a of 0 
30 and any motion metric value of 8 or more yields a blending factor a of 1 . 
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As indicated above, the blending factors a from look up table 1 1 1 are supplied to 
alpha blender 1 12 where they are employed with the field based interpolation factor from 
unit 105 and the frame based interpolation factor from unit 106. 

It has been observed, however, that alpha blending may not be required in all 
5 applications of the invention. In such situations a hard switch from frame based 
interpolation to field based interpolation is sufficient for practical results. When 
employing such hard switching from frame based interpolation to field based 
interpolation a much simplified spatial median filter can be used. This hard switching is 
readily accomplished by employing a controllable selector to select either the output from 
10 frame interpolator 106 when the image is still, e.g., a motion metric value of less than 4 
in this example, or the output from field interpolator 105 when there is motion in the 
image, i.e., a motion metric value of 4 or more in this example. 

It is noted that interpolation for chrominance is always field based. 

The above-described embodiments are, of course, merely illustrative of the 
15 principles of the invention. Indeed, numerous other methods or apparatus may be 
devised by those skilled in the art without departing from the spirit and scope of the 
invention. 
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